Identifying an availability of a system

ABSTRACT

A method according to one embodiment includes sending, from a first system to a second system, a request for a clock value associated with a third system; receiving, from the second system, a clock value associated with the third system and a query clock value determined at the second system; comparing, at the first system, the clock value associated with the third system to the query clock value determined at the second system; determining an amount of time since the third system contacted the second system, in response to determining that the query clock value is greater than the clock value associated with the third system; comparing the difference to a threshold time; determining that the third system is unavailable when the difference exceeds the predetermined threshold time value; and performing one or more predetermined actions at the first system in response to determining that the third system is unavailable.

BACKGROUND

The present invention relates to data storage and recovery, and morespecifically, this invention relates to identifying an availability orunavailability of a system in order to implement recovery procedures.

Redundant data storage is a valuable tool for maintaining data integrityand minimizing effects of storage failures. For example, data may bereplicated between two or more separate systems, and when one systemexperiences operational issues, another system may help implementrecovery operations. However, it is currently difficult to accuratelyand reliably determine an availability of each system within a redundantdata storage environment due to inconsistent, non-synchronized clockslocated at each system.

SUMMARY

A computer-implemented method according to one embodiment includessending, from a first system to a second system, a request for a clockvalue associated with a third system; receiving, from the second system,a clock value associated with the third system and a query clock valuedetermined at the second system; comparing, at the first system, theclock value associated with the third system to the query clock valuedetermined at the second system; determining an amount of time since thethird system contacted the second system, in response to determiningthat the query clock value is greater than the clock value associatedwith the third system, where the amount of time since the third systemcontacted the second system includes a difference between the clockvalue associated with the third system and the query clock value at thesecond system; comparing the difference to a predetermined thresholdtime value; determining that the third system is unavailable in responseto determining that the difference exceeds the predetermined thresholdtime value; and performing one or more predetermined actions at thefirst system in response to determining that the third system isunavailable.

According to another embodiment, a computer program product foridentifying an availability of a system includes a computer readablestorage medium having program instructions embodied therewith, where thecomputer readable storage medium is not a transitory signal per se, andwhere the program instructions are executable by a processor to causethe processor to perform a method including sending, from a first systemto a second system, a request for a clock value associated with a thirdsystem, utilizing the processor; receiving, from the second system, aclock value associated with the third system and a query clock valuedetermined at the second system, utilizing the processor; comparing,utilizing the processor at the first system, the clock value associatedwith the third system to the query clock value determined at the secondsystem; determining, utilizing the processor, an amount of time sincethe third system contacted the second system, in response to determiningthat the query clock value is greater than the clock value associatedwith the third system, where the amount of time since the third systemcontacted the second system includes a difference between the clockvalue associated with the third system and the query clock value at thesecond system; comparing the difference to a predetermined thresholdtime value, utilizing the processor; and determining, utilizing theprocessor, that the third system is unavailable in response todetermining that the difference exceeds the predetermined threshold timevalue; and performing one or more predetermined actions at the firstsystem in response to determining that the third system is unavailable,utilizing the processor.

A system according to another embodiment includes a processor; and logicintegrated with the processor, executable by the processor, orintegrated with and executable by the processor, where the logic isconfigured to send, from a first system to a second system, a requestfor a clock value associated with a third system; receive, from thesecond system, a clock value associated with the third system and aquery clock value determined at the second system; compare, at the firstsystem, the clock value associated with the third system to the queryclock value determined at the second system; determine an amount of timesince the third system contacted the second system, in response todetermining that the query clock value is greater than the clock valueassociated with the third system, where the amount of time since thethird system contacted the second system includes a difference betweenthe clock value associated with the third system and the query clockvalue at the second system; compare the difference to a predeterminedthreshold time value; determine that the third system is unavailable inresponse to determining that the difference exceeds the predeterminedthreshold time value; and perform one or more predetermined actions atthe first system in response to determining that the third system isunavailable.

Other aspects and embodiments of the present invention will becomeapparent from the following detailed description, which, when taken inconjunction with the drawings, illustrate by way of example theprinciples of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a network architecture, in accordance with oneembodiment.

FIG. 2 shows a representative hardware environment that may beassociated with the servers and/or clients of FIG. 1, in accordance withone embodiment.

FIG. 3 illustrates a tiered data storage system in accordance with oneembodiment.

FIG. 4 illustrates a method for identifying an availability of a system,in accordance with one embodiment.

FIG. 5 illustrates a method for managing a quorum witness system, inaccordance with one embodiment.

FIG. 6 illustrates a method for implementing a quorum witnessreconnection grace time, in accordance with one embodiment.

FIG. 7 illustrates a method for adjusting monotonic clock values at aquorum witness, in accordance with one embodiment.

FIG. 8 illustrates an exemplary synchronous replication environmentimplementing peer availability monitoring, in accordance with oneembodiment.

DETAILED DESCRIPTION

The following description discloses several preferred embodiments ofsystems, methods and computer program products for identifying anavailability of a system. Various embodiments provide a method toidentify an availability of a first system via clock values associatedwith the first system that are retrieved from a second system, inaddition to a query clock value retrieved from the second system.

The following description is made for the purpose of illustrating thegeneral principles of the present invention and is not meant to limitthe inventive concepts claimed herein. Further, particular featuresdescribed herein can be used in combination with other describedfeatures in each of the various possible combinations and permutations.

Unless otherwise specifically defined herein, all terms are to be giventheir broadest possible interpretation including meanings implied fromthe specification as well as meanings understood by those skilled in theart and/or as defined in dictionaries, treatises, etc.

It must also be noted that, as used in the specification and theappended claims, the singular forms “a,” “an” and “the” include pluralreferents unless otherwise specified. It will be further understood thatthe terms “includes” and/or “comprising,” when used in thisspecification, specify the presence of stated features, integers, steps,operations, elements, and/or components, but do not preclude thepresence or addition of one or more other features, integers, steps,operations, elements, components, and/or groups thereof.

The following description discloses several preferred embodiments ofsystems, methods and computer program products for identifying anavailability of a system.

In one general embodiment, a computer-implemented method includessending, from a first system to a second system, a request for a clockvalue associated with a third system, receiving, from the second system,a clock value associated with the third system and a query clock valuedetermined at the second system, comparing, at the first system, theclock value associated with the third system to the query clock valuedetermined at the second system to determine whether the third system isunavailable, and performing one or more predetermined actions at thefirst system in response to determining that the third system isunavailable.

In another general embodiment, a computer program product foridentifying an availability of a system includes a computer readablestorage medium having program instructions embodied therewith, where thecomputer readable storage medium is not a transitory signal per se, andwhere the program instructions are executable by a processor to causethe processor to perform a method comprising sending, from a firstsystem to a second system, a request for a clock value associated with athird system, utilizing the processor, receiving, from the secondsystem, a clock value associated with the third system and a query clockvalue determined at the second system, utilizing the processor,comparing, at the first system, the clock value associated with thethird system to the query clock value determined at the second system todetermine whether the third system is unavailable, utilizing theprocessor, and performing one or more predetermined actions at the firstsystem in response to determining that the third system is unavailable,utilizing the processor.

In another general embodiment, a computer-implemented method includesreceiving a message at a second system from a first system, identifyinga clock value associated with the message, storing the clock value inassociation with an identifier of the first system, receiving a query atthe second system from a third system different from the first system,and returning to the third system, by the second system, the clockvalue, the identifier of the first system, and a query clock value.

FIG. 1 illustrates an architecture 100, in accordance with oneembodiment. As shown in FIG. 1, a plurality of remote networks 102 areprovided including a first remote network 104 and a second remotenetwork 106. A gateway 101 may be coupled between the remote networks102 and a proximate network 108. In the context of the presentarchitecture 100, the networks 104, 106 may each take any formincluding, but not limited to a LAN, a WAN such as the Internet, publicswitched telephone network (PSTN), internal telephone network, etc.

In use, the gateway 101 serves as an entrance point from the remotenetworks 102 to the proximate network 108. As such, the gateway 101 mayfunction as a router, which is capable of directing a given packet ofdata that arrives at the gateway 101, and a switch, which furnishes theactual path in and out of the gateway 101 for a given packet.

Further included is at least one data server 114 coupled to theproximate network 108, and which is accessible from the remote networks102 via the gateway 101. It should be noted that the data server(s) 114may include any type of computing device/groupware. Coupled to each dataserver 114 is a plurality of user devices 116. User devices 116 may alsobe connected directly through one of the networks 104, 106, 108. Suchuser devices 116 may include a desktop computer, lap-top computer,hand-held computer, printer or any other type of logic. It should benoted that a user device 111 may also be directly coupled to any of thenetworks, in one embodiment.

A peripheral 120 or series of peripherals 120, e.g., facsimile machines,printers, networked and/or local storage units or systems, etc., may becoupled to one or more of the networks 104, 106, 108. It should be notedthat databases and/or additional components may be utilized with, orintegrated into, any type of network element coupled to the networks104, 106, 108. In the context of the present description, a networkelement may refer to any component of a network.

According to some approaches, methods and systems described herein maybe implemented with and/or on virtual systems and/or systems whichemulate one or more other systems, such as a UNIX system which emulatesan IBM z/OS environment, a UNIX system which virtually hosts a MICROSOFTWINDOWS environment, a MICROSOFT WINDOWS system which emulates an IBMz/OS environment, etc. This virtualization and/or emulation may beenhanced through the use of VMWARE software, in some embodiments.

In more approaches, one or more networks 104, 106, 108, may represent acluster of systems commonly referred to as a “cloud.” In cloudcomputing, shared resources, such as processing power, peripherals,software, data, servers, etc., are provided to any system in the cloudin an on-demand relationship, thereby allowing access and distributionof services across many computing systems. Cloud computing typicallyinvolves an Internet connection between the systems operating in thecloud, but other techniques of connecting the systems may also be used.

FIG. 2 shows a representative hardware environment associated with auser device 116 and/or server 114 of FIG. 1, in accordance with oneembodiment. Such figure illustrates a typical hardware configuration ofa workstation having a central processing unit 210, such as amicroprocessor, and a number of other units interconnected via a systembus 212.

The workstation shown in FIG. 2 includes a Random Access Memory (RAM)214, Read Only Memory (ROM) 216, an I/O adapter 218 for connectingperipheral devices such as disk storage units 220 to the bus 212, a userinterface adapter 222 for connecting a keyboard 224, a mouse 226, aspeaker 228, a microphone 232, and/or other user interface devices suchas a touch screen and a digital camera (not shown) to the bus 212,communication adapter 234 for connecting the workstation to acommunication network 235 (e.g., a data processing network) and adisplay adapter 236 for connecting the bus 212 to a display device 238.

The workstation may have resident thereon an operating system such asthe Microsoft Windows® Operating System (OS), a MAC OS, a UNIX OS, etc.It will be appreciated that a preferred embodiment may also beimplemented on platforms and operating systems other than thosementioned. A preferred embodiment may be written using XML, C, and/orC++ language, or other programming languages, along with an objectoriented programming methodology. Object oriented programming (OOP),which has become increasingly used to develop complex applications, maybe used.

Now referring to FIG. 3, a storage system 300 is shown according to oneembodiment. Note that some of the elements shown in FIG. 3 may beimplemented as hardware and/or software, according to variousembodiments. The storage system 300 may include a storage system manager312 for communicating with a plurality of media on at least one higherstorage tier 302 and at least one lower storage tier 306. The higherstorage tier(s) 302 preferably may include one or more random accessand/or direct access media 304, such as hard disks in hard disk drives(HDDs), nonvolatile memory (NVM), solid state memory in solid statedrives (SSDs), flash memory, SSD arrays, flash memory arrays, etc.,and/or others noted herein or known in the art. The lower storagetier(s) 306 may preferably include one or more lower performing storagemedia 308, including sequential access media such as magnetic tape intape drives and/or optical media, slower accessing HDDs, sloweraccessing SSDs, etc., and/or others noted herein or known in the art.One or more additional storage tiers 316 may include any combination ofstorage memory media as desired by a designer of the system 300. Also,any of the higher storage tiers 302 and/or the lower storage tiers 306may include some combination of storage devices and/or storage media.

The storage system manager 312 may communicate with the storage media304, 308 on the higher storage tier(s) 302 and lower storage tier(s) 306through a network 310, such as a storage area network (SAN), as shown inFIG. 3, or some other suitable network type. The storage system manager312 may also communicate with one or more host systems (not shown)through a host interface 314, which may or may not be a part of thestorage system manager 312. The storage system manager 312 and/or anyother component of the storage system 300 may be implemented in hardwareand/or software, and may make use of a processor (not shown) forexecuting commands of a type known in the art, such as a centralprocessing unit (CPU), a field programmable gate array (FPGA), anapplication specific integrated circuit (ASIC), etc. Of course, anyarrangement of a storage system may be used, as will be apparent tothose of skill in the art upon reading the present description.

In more embodiments, the storage system 300 may include any number ofdata storage tiers, and may include the same or different storage memorymedia within each storage tier. For example, each data storage tier mayinclude the same type of storage memory media, such as HDDs, SSDs,sequential access media (tape in tape drives, optical disk in opticaldisk drives, etc.), direct access media (CD-ROM, DVD-ROM, etc.), or anycombination of media storage types. In one such configuration, a higherstorage tier 302, may include a majority of SSD storage media forstoring data in a higher performing storage environment, and remainingstorage tiers, including lower storage tier 306 and additional storagetiers 316 may include any combination of SSDs, HDDs, tape drives, etc.,for storing data in a lower performing storage environment. In this way,more frequently accessed data, data having a higher priority, dataneeding to be accessed more quickly, etc., may be stored to the higherstorage tier 302, while data not having one of these attributes may bestored to the additional storage tiers 316, including lower storage tier306. Of course, one of skill in the art, upon reading the presentdescriptions, may devise many other combinations of storage media typesto implement into different storage schemes, according to theembodiments presented herein.

According to some embodiments, the storage system (such as 300) mayinclude logic configured to receive a request to open a data set, logicconfigured to determine if the requested data set is stored to a lowerstorage tier 306 of a tiered data storage system 300 in multipleassociated portions, logic configured to move each associated portion ofthe requested data set to a higher storage tier 302 of the tiered datastorage system 300, and logic configured to assemble the requested dataset on the higher storage tier 302 of the tiered data storage system 300from the associated portions.

Of course, this logic may be implemented as a method on any deviceand/or system or as a computer program product, according to variousembodiments.

Now referring to FIG. 4, a flowchart of a method 400 is shown accordingto one embodiment. The method 400 may be performed in accordance withthe present invention in any of the environments depicted in FIGS. 1-3and 8, among others, in various embodiments. Of course, more or lessoperations than those specifically described in FIG. 4 may be includedin method 400, as would be understood by one of skill in the art uponreading the present descriptions.

Each of the steps of the method 400 may be performed by any suitablecomponent of the operating environment. For example, in variousembodiments, the method 400 may be partially or entirely performed byone or more servers, computers, or some other device having one or moreprocessors therein. The processor, e.g., processing circuit(s), chip(s),and/or module(s) implemented in hardware and/or software, and preferablyhaving at least one hardware component may be utilized in any device toperform one or more steps of the method 400. Illustrative processorsinclude, but are not limited to, a central processing unit (CPU), anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA), etc., combinations thereof, or any other suitablecomputing device known in the art.

As shown in FIG. 4, method 400 may initiate with operation 402, where arequest for a clock value associated with a third system is sent from afirst system to a second system. In one embodiment, the first system andthe third system may each include a storage array. For example, eachstorage array may comprise a plurality of storage volumes for storingdata. In another embodiment, the first system may contain a volume thatis synchronously replicated with a volume that is stored on the thirdsystem (e.g., to create a synchronously replicated stretch volume,etc.). Of course, however, the first system and third system may eachinclude any type of computerized system.

Additionally, in one embodiment, the second system may include a quorumwitness. For example, the quorum witness may include a system that is incommunication with, and maintains a status of, each of a plurality ofother systems (e.g., at least the first system and the third system,etc.). In another embodiment, the request for the clock value mayinclude a query. In yet another embodiment, the first system may sendthe query to the second system using a node (e.g., a quorum node, etc.)within the first system.

Further, in one embodiment, the first, second, and third systems may allbe physically separate from each other. For example, each system may beimplemented using different hardware from the other systems. In anotherexample, each system may be located at a different physical locationfrom the other systems.

Further still, in one embodiment, the request may include a query to thesecond system. In another embodiment, the clock value associated withthe third system may include a clock value associated with a statusnotification last received by the second system by the third system. Inyet another embodiment, the request may include a general request forclock values associated with all systems in communication with thesecond system.

Also, method 400 may proceed with operation 404, where a clock valueassociated with the third system and a query clock value determined atthe second system is received from the second system. In one embodiment,the clock value associated with the third system and the query clockvalue determined at the second system may be received from the secondsystem in response to the request.

In addition, in one embodiment, the query clock value may include avalue for a clock at the second system when the request (e.g., for theclock value associated with the third system) was received from thefirst system at the second system. For example, the clock may include amonotonic clock. In another example, the monotonic clock may include aclock that starts at a random start point (e.g., in response to astart-up of the second system, etc.). In yet another example, themonotonic clock may be incremented monotonically (e.g., withoutvariation, etc.) from the random start point. In this way, the monotonicclock may act as a timer that is unaffected by time changes (e.g., timechanges made to a system clock of the second system, etc.).

Furthermore, in one embodiment, the clock value associated with thethird system may include a value for the clock at the second system whenthe third system last contacted the second system. For example, thirdsystem may periodically contact the second system by sending a messageto the second system (e.g., at a predetermined interval, etc.). Inanother example, the message may include a heartbeat message (e.g., amessage sent from a first entity to a second entity that enables thesecond entity or another entity to identify if and when the first entityfails or is no longer available, etc.).

Further still, in one embodiment, the clock value associated with thethird system and the query clock value may be adjusted to account for arandom starting point of a monotonic clock of the second system. Inanother embodiment, the first system may receive values for the clock atthe second system when each of a plurality of systems last contacted thesecond system. For example, all clock values returned by the secondsystem may be adjusted by the second system prior to returning the clockvalues to account for a random starting point of a monotonic clock ofthe second system.

Also, method 400 may proceed with operation 406, where the clock valueassociated with the third system is compared at the first system to thequery clock value determined at the second system to determine whetherthe third system is unavailable. In one embodiment, the comparison mayresult in determining that the query clock value is greater than theclock value associated with the third system.

Additionally, in one embodiment, an amount of time since the thirdsystem contacted the second system may be determined, in response todetermining that the query clock value is greater than the clock valueassociated with the third system. For example, the amount of time sincethe third system contacted the second system may include a differencebetween the clock value associated with the third system and the queryclock value at the second system.

Further, in one embodiment, the difference may be compared to apredetermined threshold time value (e.g., three seconds, etc.). Inanother embodiment, it may be determined that the third system isunavailable in response to determining that the difference exceeds thepredetermined threshold time value. In yet another embodiment, it may bedetermined that the third system is available in response to determiningthat the difference does not exceed the predetermined threshold timevalue.

Further still, in one embodiment, the comparison may result indetermining that the query clock value is less than the clock valueassociated with the third system. In another embodiment, the clock valueassociated with the third system may be compared to a predeterminedreconnection grace time, in response to determining that the query clockvalue is less than the clock value associated with the third system.

Also, in one embodiment, it may be determined that the third system isunavailable in response to determining that the clock value associatedwith the third system is greater than a predetermined reconnection gracetime. In another embodiment, no determination may be made about thethird system in response to determining that the clock value associatedwith the third system is less than the predetermined reconnection gracetime. In yet another embodiment, the reconnection grace time may includea predetermined time value (e.g., ten seconds, etc.) associated with atime for the second system to restart and reset its clock.

In addition, method 400 may proceed with operation 408, where one ormore predetermined actions are performed at the first system in responseto determining that the third system is unavailable. In one embodiment,the one or more predetermined actions may include one or more fail-overoperations, one or more fail-back operations, one or more recoveryoperations, etc.

For example, the first system may include a volume that is synchronouslyreplicated on the third system (e.g., to create a synchronouslyreplicated stretch volume, etc.). In another embodiment, thesynchronously replicated volume may be provided to one or more hosts,and one or more applications running on the one or more hosts. In yetanother embodiment, before becoming unavailable, the third system mayprovide all or a portion of requested data from the synchronouslyreplicated volume to the one or more hosts and the one or moreapplications running on the one or more hosts.

In another example, in response to determining that the third system isunavailable, the first system may provide all requested data from thesynchronously replicated volume to the one or more hosts and the one ormore applications running on the one or more hosts (e.g., instead of thethird system, etc.). In yet another example, the first system may alsobe used to restore the synchronously replicated volume at the thirdsystem, in response to determining that the third system is unavailable.

Furthermore, in one embodiment, the third system may be labeled asunavailable in response to determining that the third system isunavailable. In another embodiment, one or more notifications may besent (e.g., to one or more users, administrators, etc.) in response todetermining that the third system is unavailable.

In this way, accurate system availability may be determined betweensystems without the need to synchronize inconsistent clocks within thesystems. For example, clock values for the first system and the secondsystem need not be stored in persistent memory in order to determinesystem availability. This may reduce an amount of data that needs to bestored in order to determine system availability, and may also reduce anamount of processing that needs to be performed in order to determinesystem availability (e.g., since only a comparison between the clockvalue associated with the third system and the query clock valuedetermined at the second system is necessary to determine anavailability of the third system).

Now referring to FIG. 5, a flowchart of a method 500 for managing aquorum witness system is shown according to one embodiment. The method500 may be performed in accordance with the present invention in any ofthe environments depicted in FIGS. 1-3 and 8, among others, in variousembodiments. Of course, more or less operations than those specificallydescribed in FIG. 5 may be included in method 500, as would beunderstood by one of skill in the art upon reading the presentdescriptions.

Each of the steps of the method 500 may be performed by any suitablecomponent of the operating environment. For example, in variousembodiments, the method 500 may be partially or entirely performed byone or more servers, computers, or some other device having one or moreprocessors therein. The processor, e.g., processing circuit(s), chip(s),and/or module(s) implemented in hardware and/or software, and preferablyhaving at least one hardware component may be utilized in any device toperform one or more steps of the method 500. Illustrative processorsinclude, but are not limited to, a central processing unit (CPU), anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA), etc., combinations thereof, or any other suitablecomputing device known in the art.

As shown in FIG. 5, method 500 may initiate with operation 502, where amessage is received at a second system from a first system. In oneembodiment, the first system may include a storage array. In anotherembodiment, the second system may include a quorum witness. In yetanother embodiment, the message may include a status notification (e.g.,a “heartbeat” notification, etc.).

Additionally, method 500 may proceed with operation 504, where a clockvalue associated with the message is identified. In one embodiment, theclock value may include a time value indicated by a clock at the secondsystem. In another embodiment, the clock value may include a timeindicated by the clock at the second system when the message wasreceived at the second system from the first system. In yet anotherembodiment, the clock may include a monotonic clock. For example, theclock value may be adjusted by the second system to account for a randomstarting point of a monotonic clock of the second system.

Further, method 500 may proceed with operation 506, where the clockvalue is stored in association with an identifier of the first system.For example, the clock value may be stored at the second system (e.g.,as a timestamp). In another example, the clock value may be stored in adatabase of timestamps of the second system. In yet another example, theclock value may be stored in association with an identifier of the firstsystem.

Further still, in one embodiment, the clock value may overwrite anearlier stored clock value in association with the identifier of thefirst system. For example, the clock value may be stored as a lastreceived status notification from the first system.

Also, method 500 may proceed with operation 508, where a query isreceived at the second system from a third system different from thefirst system. In one embodiment, the third system may also include astorage array. In another embodiment, the third system may have astorage volume that is replicated (e.g., synchronously) on the firstsystem. In yet another embodiment, the query may include a query for aclock value associated with all systems other than the third system thatare in communication with the second system. In still anotherembodiment, the query may include a query for a clock value associatedwith a specific system.

In addition, method 500 may proceed with operation 510, where the clockvalue, the identifier of the first system, and a query clock value arereturned to the third system by the second system. In one embodiment,the query clock value may include a value for a clock at the secondsystem when the request was received from the third system at the secondsystem. In another embodiment, the query clock value may be returnedwith an associated identifier (e.g., identifying the query clock valueas the time the query was received, etc.). In yet another embodiment,query clock values for all systems in communication with the secondsystem may be returned. For example, each clock value may have anassociated identifier indicating the specific system having that clockvalue.

Now referring to FIG. 6, a flowchart of a method 600 for implementing aquorum witness reconnection grace time is shown according to oneembodiment. The method 600 may be performed in accordance with thepresent invention in any of the environments depicted in FIGS. 1-3 and8, among others, in various embodiments. Of course, more or lessoperations than those specifically described in FIG. 6 may be includedin method 600, as would be understood by one of skill in the art uponreading the present descriptions.

Each of the steps of the method 600 may be performed by any suitablecomponent of the operating environment. For example, in variousembodiments, the method 600 may be partially or entirely performed byone or more servers, computers, or some other device having one or moreprocessors therein. The processor, e.g., processing circuit(s), chip(s),and/or module(s) implemented in hardware and/or software, and preferablyhaving at least one hardware component may be utilized in any device toperform one or more steps of the method 600. Illustrative processorsinclude, but are not limited to, a central processing unit (CPU), anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA), etc., combinations thereof, or any other suitablecomputing device known in the art.

As shown in FIG. 6, method 600 may initiate with operation 602, where aquery for a clock value associated with a third system is sent from afirst system to a second system. In one embodiment, the first and thirdsystems may each include a distinct storage array. In anotherembodiment, the second system may include a quorum witness.

Additionally, method 600 may proceed with operation 604, where a clockvalue associated with the third system and a query clock valuedetermined at the second system are received at the first system fromthe second system. In one embodiment, the clock value associated withthe third system may include a time indicated by the clock at the secondsystem when a status notification message was received at the secondsystem from the third system.

Further, method 600 may proceed with operation 606, where in response todetermining that the clock value associated with a third system isgreater than the query clock value determined at the second system, oneor more predetermined actions are performed at the first system inresponse to determining that the query clock value determined at thesecond system is greater than a reconnection grace time.

Further still, in one embodiment, the one or more predetermined actionsmay include one or more fail-over operations. In another embodiment, theone or more predetermined actions may include labelling the third systemas unavailable. In yet another embodiment, the reconnection grace timemay include a predetermined time value (e.g., ten seconds, etc.)associated with a time for the second system to restart and reset itsclock.

Now referring to FIG. 7, a flowchart of a method 700 for adjustingmonotonic clock values at a quorum witness is shown according to oneembodiment. The method 700 may be performed in accordance with thepresent invention in any of the environments depicted in FIGS. 1-3 and8, among others, in various embodiments. Of course, more or lessoperations than those specifically described in FIG. 7 may be includedin method 700, as would be understood by one of skill in the art uponreading the present descriptions.

Each of the steps of the method 700 may be performed by any suitablecomponent of the operating environment. For example, in variousembodiments, the method 700 may be partially or entirely performed byone or more servers, computers, or some other device having one or moreprocessors therein. The processor, e.g., processing circuit(s), chip(s),and/or module(s) implemented in hardware and/or software, and preferablyhaving at least one hardware component may be utilized in any device toperform one or more steps of the method 700. Illustrative processorsinclude, but are not limited to, a central processing unit (CPU), anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA), etc., combinations thereof, or any other suitablecomputing device known in the art.

As shown in FIG. 7, method 700 may initiate with operation 702, where astarting clock value associated with a clock is identified at a secondsystem. In one embodiment, the starting clock value may be identified inresponse to a loading of the second system. In another embodiment, thesecond system may include a quorum witness. In yet another embodiment,the clock may include a monotonic clock.

Additionally, method 700 may proceed with operation 704, where a messagefrom a first system is received at the second system. In one embodiment,the first system may include a storage array. In another embodiment, themessage may include a status notification.

Further, method 700 may proceed with operation 706, where a clock valueassociated with the message is identified, where the clock valueincludes a value of the clock at the second system when the message isreceived at the second system from the first system. Further still,method 700 may proceed with operation 708, where the starting clockvalue is subtracted from the clock value associated with the message toobtain an adjusted clock value associated with the message. This mayadjust the clock value associated with the message to account for arandom starting point of a monotonic clock of the second system.

Further still, method 700 may proceed with operation 710, where theadjusted clock value is stored at the second system in association withan identifier of the first system. Also, method 700 may proceed withoperation 712, where a query is received at the second system from athird system different from the first system. In one embodiment, thethird system may include a storage array. In another embodiment, thefirst system may contain a volume that is synchronously replicated witha volume that is contained within the third system (e.g., to create asynchronously replicated stretch volume, etc.).

In addition, method 700 may proceed with operation 714, where a queryclock value is identified at the second system when the query wasreceived at the second system. For example, may a value of a monotonicclock at the second system may be identified by the second system.

Furthermore, method 700 may proceed with operation 716, where thestarting clock value is subtracted from the query clock value at thesecond system to obtain an adjusted query clock value. Further still,method 700 may proceed with operation 718, where the adjusted clockvalue, the identifier of the first system, and the adjusted query clockvalue are returned to the third system.

In this way, all time stamps returned by the quorum witness may beadjusted to account for a random starting point of a monotonic clock ofthe quorum witness, such that all adjusted time stamps start at a timet=0.

FIG. 8 illustrates a synchronous replication environment 800implementing peer availability monitoring, according to one embodiment.As shown in FIG. 8, the environment 800 includes a plurality of storagearrays 802A-N that are each in communication with a quorum witness 804via respective quorum nodes 806A-N within each of the storage arrays802A-N.

Additionally, each of the storage arrays 802A-N includes a plurality ofstorage volumes 808A-N and 810A-N, respectively. In one embodiment, oneof the storage volumes 808A-N within the first storage array 802A may besynchronously replicated as another one of the storage volumes 810A-N ofthe Nth storage array 802N. For example, a first storage volume 808A anda second storage volume 810A may be replicated synchronously to create astretch volume. In another embodiment, each of the storage arrays 802A-Nmay include separate hardware, may be located at a different locationfrom the other storage arrays, etc.

Further, in one embodiment, each of the storage arrays 802A-N mayperiodically send a status notification message to the quorum witness804. For example, each of the storage arrays 802A-N may send a statusnotification message to the quorum witness 804 at a predetermined timeinterval. Upon receiving a status notification message (e.g., a“heartbeat” message, etc.) from one of the storage arrays 802A-N, thequorum witness 804 may identify a clock value indicated by a monotonicclock 812 of the quorum witness 804, and may record the clock value inassociation with the one of the storage arrays 802A-N that sent thestatus notification message (e.g., in a database of the quorum witness804, etc.).

In another embodiment, the recorded time may overwrite a previouslyrecorded time for the one of the storage arrays 802A-N. In yet anotherembodiment, the quorum witness 804 may adjust the recorded time toaccount for a random starting point of the monotonic clock 812 of thequorum witness 804. In this way, the quorum witness 804 may store clockvalues indicating a monotonic clock time at which the quorum witnessmost recently received status notification messages from each of thestorage arrays 802A-N.

Further still, in one embodiment, one of the storage arrays 802A-N maysend a query (e.g., utilizing their respective quorum node 806A-N, etc.)to the quorum witness 804. For example, the query may include a requestfor clock values at which the quorum witness most recently receivedstatus notification messages from each of the storage arrays 802A-N. Inresponse to the query, the quorum witness 804 may first identify a queryclock value indicated by the monotonic clock 812 of the quorum witness804 at which the query was received.

The quorum witness may then send to the requesting one of the storagearrays 802A-N the query clock value, as well as clock values at whichthe quorum witness most recently received status notification messagesfrom each of the storage arrays 802A-N. Upon receiving such information,the requesting one of the storage arrays 802A-N may determine which ofthe storage arrays 802A-N are unavailable, and may perform one or moreassociated fail-over or fail-back actions.

Identifying an Unhealthy System According to Timestamp and Monotime

One goal of a transparent failover/hyperswap feature may includesupporting a high availability or recovery time objective (RTO) of zero.After an event that destroys production storage, the time to recover andmake applications operational is zero. For example, the applicationshould experience no down time as a result of a storage failure.

A stretch cluster is a high availability (HA) solution that may enablehosts to access a stretch volume, which may be made of a set of twovolumes replicated synchronously that reside on two separate storagearrays. The stretch volume may appear as a single volume to the hostsand may make transitions such as fail-over and fail-back completelytransparent to the hosts and the applications running on the hosts.

Table 1 illustrates exemplary components of an HA solution, inaccordance with one embodiment. Of course, it should be noted that theexemplary components shown in Table 1 are set forth for illustrativepurposes only, and thus should not be construed as limiting in anymanner.

TABLE 1 COMPONENT DESCRIPTION Stretch volume The volumes' contents areidentical by synchronous replication SCSI behavior consistent with asingle volume exposed through separate paths: The volumes have the sameSCSI identity through Volume masquerading The volumes are consistentwith regards to SCSI reservations Active-preferred/active- AsymmetricLogical Unit Access (ALUA) non-preferred access allows paths to a SCSIdevice to be marked as having distinct characteristics. ALUA will beused to export the stretch volume from both storage arrayssimultaneously, the paths to the storage array holding the Master volumewill be marked as active/preferred and the paths to the Slave sidevolume will be marked as active/non-preferred HA Logic HA Logic: Logicwhich identifies Master failure and triggers automatic fail-over AQuorum Witness (QW) at a third location with connectivity to bothsystems will be used as a tie breaker A Quorum node (QN) on each storagearray communicates with the QW

To fulfill the transparent failover/hyperswap feature, there are manyactors that may need to be in sync—the QW and each storage arrayconnection, via a QN, to the QW. All these actors may coordinatefail-overs, based on a peer storage array going down, and they may do soas fast as possible without false-positives.

The time since the last communication between the remote peer and the QWis used to determine a need for fail-overs. This may be used togetherwith local peer knowledge to determine whether the remote peer isavailable or not. Solely relying on the local peer knowledge may beavoided, as direct communication between the storage arrays may besevered and any local peer knowledge may be outdated.

Each one of the actors has an independent clock, which itself may bemoved by user intervention or automatically based on Network TimeProtocol (NTP) updates (e.g. by moving to daylight savings, etc.). As aresult, there is a need for all the actors to properly detect andcommunicate peer availability, even though their clocks are notconsistent and are not synchronized.

In one embodiment, all the actors may report changes by utilizing onlythe monotonic clock of the QW, with QW restart grace timeconsiderations. Each request by any storage array may not contain anyclock information, and responses from the QW may contain the QW'smonotonic clock's value at the time of request processing. These valuesmay be the only values used as reference in any determinations of peeravailability by the storage arrays.

This solution may rely solely on the monotonic clock of the QW. Thisclock may not be affected by NTP updates and may not be able to bechanged via user intervention. The monotonic clock may receive aninitial value upon system power-up and may consistently increase itsvalue. Each storage array may register on the QW and the QW may create arecord of the storage array for monitoring its latest “I'm still alive”announcement, or heartbeat (HB).

After the storage arrays register, they send HBs during a predeterminedinterval (e.g., every 1 second, etc.) to the QW. These HBs may beprocessed by the QW and the latter may store the monotonic clock valueat the time of processing, or time-stamp, translated to milliseconds, asthe last HB time of the storage array.

When other storage arrays query for peer storage array HB times, the QWmay add the current monotonic clock value at the time of the queryprocessing, or query time, to the query response.

Table 2 illustrates an exemplary query response from the QW, inaccordance with one embodiment. Of course, it should be noted that theexemplary query response shown in Table 2 is set forth for illustrativepurposes only, and thus should not be construed as limiting in anymanner.

TABLE 2 { “query_time” : 216515132, “systems” : [ { “storage_array_id” :“storage_array_1”, “HB_time” : 142240924 }, { “storage_array_id” :“storage_array_2”, “HB_time” : 216515059 }, { “storage_array_id” :“storage_array_3”, “HB_time” : 216515129 }, { “storage_array_id” :“storage_array_4”, “HB_time” : 216515118 }, { “storage_array_id” :“storage_array_5”, “HB_time” : 322515117 }, { “storage_array_id” :“storage_array_6”, “HB_time” : 215891880 }, { “storage_array_id” :“storage_array_7”, “HB_time” : 6386 }, { “storage_array_id” :“storage_array_8”, “HB_time” : 6386 }, { “storage_array_id” :“storage_array_9”, “HB_time” : 216514992 } ] }

The querying storage array may now compare each storage array's HB timewith the query time, as they are both values from the same clock. Forexample, if the peer storage array's HB time is smaller than the querytime by more than a predetermined time (e.g., 3 seconds, etc.), then thepeer storage array has stopped communicating with the QW and isconsidered unavailable. In the sample response in Table 2, storagearrays 1, 6, 7 and 8 may all be considered unavailable, as 3 seconds is3000 milliseconds and each of these storage arrays' HB time value issmaller than the query time value by more than 3,000 milliseconds.

In another embodiment, if the peer storage array's HB time is notsmaller than the query time by more than the predetermined time (e.g.,the peer storage array's HB time is smaller than the query time but byless than 3 seconds, etc.) then the system may be considered available.In the sample response in Table 2, storage arrays 2, 3, 4 and 9 may allbe considered available, as each of these storage arrays' HB time valueis smaller than the query time value by less than 3,000 milliseconds.

Additionally, in one embodiment, if the machine running the QW isrestarted, the monotonic clock may be reset. In such a case, the storagearrays' HB times may possibly be greater than the query time. For thiswe introduce a reconnection grace time. For example, if the peer storagearray's HB time is smaller than the query time, the peer storage array'sHB time may be checked as mentioned above. However, if the peer storagearray's HB time is greater than the query time, the query time may beexamined.

For example, if the query time is smaller than the reconnection gracetime, then no conclusion may be made about the peer storage array'sstate. In another example, if the query time is greater than thereconnection grace time, the peer storage array may be consideredunavailable, as it should have reconnected within the reconnection gracetime.

A predetermined grace time (e.g., 10 seconds, etc.) may be set, since ittakes some time for the QW to restart all of its components, and a fewseconds may pass between setting the initial QW time-stamp from themonotonic clock value and until the first connection from a storagesystem can be processed. In the sample response in Table 2, storagearray 5 may be considered unavailable, as the storage array's HB time isgreater than the query time and the query time is greater than thereconnection grace time of 10 seconds, i.e. 10,000 milliseconds.

In one embodiment, a monotonic clock cannot be set and representsmonotonic time since a random (e.g., unspecified, etc.) starting point.In view of this, it may be confirmed that all time-stamps used by theQW, as taken from the monotonic clock, start from zero. Otherwise, theremay be no way of determining when the grace time has passed.

In order to accomplish this, when the QW loads, it may take themonotonic clock time at some moment (e.g., as a “QW start timestamp,”etc.). From that moment, all values based on the monotonic clock mayhave the QW start timestamp value subtracted from them. For instance,this value may be subtracted from all storage array HB times and fromall query times.

Table 3 illustrates an exemplary summary depicting what each storagearray concludes about peer storage arrays' availability, based on the QWquery response, in accordance with one embodiment. Of course, it shouldbe noted that the exemplary summary shown in Table 3 is set forth forillustrative purposes only, and thus should not be construed as limitingin any manner.

TABLE 3 Query time before Query time after reconnection gracereconnection grace Query response period period Peer storage array HBN/A Unavailable time > Query time Peer storage array HB UnavailableUnavailable time < Query time − threshold time value Query time −threshold Available Available time value ≤ Peer storage array HB time ≤Query time

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein includes anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which includes one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

Moreover, a system according to various embodiments may include aprocessor and logic integrated with and/or executable by the processor,the logic being configured to perform one or more of the process stepsrecited herein. By integrated with, what is meant is that the processorhas logic embedded therewith as hardware logic, such as an applicationspecific integrated circuit (ASIC), a FPGA, etc. By executable by theprocessor, what is meant is that the logic is hardware logic; softwarelogic such as firmware, part of an operating system, part of anapplication program; etc., or some combination of hardware and softwarelogic that is accessible by the processor and configured to cause theprocessor to perform some functionality upon execution by the processor.Software logic may be stored on local and/or remote memory of any memorytype, as known in the art. Any processor known in the art may be used,such as a software processor module and/or a hardware processor such asan ASIC, a FPGA, a central processing unit (CPU), an integrated circuit(IC), a graphics processing unit (GPU), etc.

It will be clear that the various features of the foregoing systemsand/or methodologies may be combined in any way, creating a plurality ofcombinations from the descriptions presented above.

It will be further appreciated that embodiments of the present inventionmay be provided in the form of a service deployed on behalf of acustomer to offer service on demand.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. Thus, the breadth and scope of a preferred embodiment shouldnot be limited by any of the above-described exemplary embodiments, butshould be defined only in accordance with the following claims and theirequivalents.

What is claimed is:
 1. A computer-implemented method, comprising:sending, from a first system to a second system, a request for a clockvalue associated with a third system; receiving, from the second system,a clock value associated with the third system and a query clock valuedetermined at the second system; comparing, at the first system, theclock value associated with the third system to the query clock valuedetermined at the second system; determining an amount of time since thethird system contacted the second system, in response to determiningthat the query clock value is greater than the clock value associatedwith the third system, where the amount of time since the third systemcontacted the second system includes a difference between the clockvalue associated with the third system and the query clock value at thesecond system; comparing the difference to a predetermined thresholdtime value; determining that the third system is unavailable in responseto determining that the difference exceeds the predetermined thresholdtime value; and performing one or more predetermined actions at thefirst system in response to determining that the third system isunavailable.
 2. The computer-implemented method of claim 1, wherein thefirst system and the third system each include a storage array.
 3. Thecomputer-implemented method of claim 1, wherein the second systemincludes a quorum witness that is in communication with, and maintains astatus of, at least the first system and the third system.
 4. Thecomputer-implemented method of claim 1, wherein the first system sendsthe request to the second system using a quorum node within the firstsystem.
 5. The computer-implemented method of claim 1, wherein therequest includes a general request for clock values associated with allsystems in communication with the second system.
 6. Thecomputer-implemented method of claim 1, wherein the query clock valueincludes a value for a clock at the second system when the request forthe clock value associated with the third system was received from thefirst system at the second system.
 7. The computer-implemented method ofclaim 1, wherein a clock at the second system includes a monotonicclock.
 8. The computer-implemented method of claim 1, wherein the clockvalue associated with the third system includes a value for a clock atthe second system when the third system last contacted the secondsystem.
 9. The computer-implemented method of claim 1, wherein the clockvalue associated with the third system and the query clock value areadjusted to account for a random starting point of a monotonic clock ofthe second system.
 10. The computer-implemented method of claim 1,further comprising: comparing the clock value associated with the thirdsystem to a predetermined reconnection grace time, in response todetermining that the query clock value is less than the clock valueassociated with the third system; and determining that the third systemis unavailable in response to determining that the clock valueassociated with the third system is greater than the predeterminedreconnection grace time.
 11. The computer-implemented method of claim 1,wherein the one or more predetermined actions include one or morefail-over operations.
 12. A computer program product for identifying anavailability of a system, the computer program product comprising acomputer readable storage medium having program instructions embodiedtherewith, wherein the computer readable storage medium is not atransitory signal per se, the program instructions executable by aprocessor to cause the processor to perform a method comprising:sending, from a first system to a second system, a request for a clockvalue associated with a third system, utilizing the processor;receiving, from the second system, a clock value associated with thethird system and a query clock value determined at the second system,utilizing the processor; comparing, utilizing the processor at the firstsystem, the clock value associated with the third system to the queryclock value determined at the second system; determining, utilizing theprocessor, an amount of time since the third system contacted the secondsystem, in response to determining that the query clock value is greaterthan the clock value associated with the third system, where the amountof time since the third system contacted the second system includes adifference between the clock value associated with the third system andthe query clock value at the second system; comparing the difference toa predetermined threshold time value, utilizing the processor; anddetermining, utilizing the processor, that the third system isunavailable in response to determining that the difference exceeds thepredetermined threshold time value; and performing one or morepredetermined actions at the first system in response to determiningthat the third system is unavailable, utilizing the processor.
 13. Thecomputer program product of claim 12, wherein the first system and thethird system each include a storage array.
 14. The computer programproduct of claim 12, wherein the second system includes a quorum witnessthat is in communication with, and maintains a status of, at least thefirst system and the third system.
 15. The computer program product ofclaim 12, wherein the first system sends the request to the secondsystem using a quorum node within the first system.
 16. The computerprogram product of claim 12, wherein the request includes a generalrequest for clock values associated with all systems in communicationwith the second system.
 17. The computer program product of claim 12,wherein the query clock value includes a value for a clock at the secondsystem when the request for the clock value associated with the thirdsystem was received from the first system at the second system.
 18. Thecomputer program product of claim 12, wherein a clock at the secondsystem includes a monotonic clock.
 19. The computer program product ofclaim 12, wherein the clock value associated with the third systemincludes a value for a clock at the second system when the third systemlast contacted the second system.
 20. A system, comprising: a processor;and logic integrated with the processor, executable by the processor, orintegrated with and executable by the processor, the logic beingconfigured to: send, from a first system to a second system, a requestfor a clock value associated with a third system; receive, from thesecond system, a clock value associated with the third system and aquery clock value determined at the second system; compare, at the firstsystem, the clock value associated with the third system to the queryclock value determined at the second system; determine an amount of timesince the third system contacted the second system, in response todetermining that the query clock value is greater than the clock valueassociated with the third system, where the amount of time since thethird system contacted the second system includes a difference betweenthe clock value associated with the third system and the query clockvalue at the second system; compare the difference to a predeterminedthreshold time value; determine that the third system is unavailable inresponse to determining that the difference exceeds the predeterminedthreshold time value; and perform one or more predetermined actions atthe first system in response to determining that the third system isunavailable.